The whole bulk load download process is like:

1. meta server set app and all partitions bulk load status is downloading feat(bulk-load): meta server start bulk load #454
2. meta server send request to primary to download files feat(bulk-load): meta server send bulk load request #457
3. primary download files
4. primary broadcast request to secondary feat(bulk-load): group bulk load request in replica group #460
5. secondary download files
6. secondary report download progress and status to primary
7. primary report group download progress and status to meta server
8. meta receive download progress
   • if all replicas of the group finish download all files, meta set this partition status as downloaded, and if all partitions are downloaded, app status will be set as downloaded, otherwise, meta will resend request to primary.

This pull request is part one of replica download files, step 3 and step 5 above.

• add an option called max_concurrent_bulk_load_downloading_count to restrict the replica count executing bulk load downloading on the node (replica_stub), and this option call be configured by remote command.
• bulk_load_start_download: if current downloading count does not exceed restriction, clear context and call download_sst_files
  • primary_context::cleanup_bulk_load_states will cleanup bulk load states reported from secondaries
  • replica::clear_bulk_load_states will cleanup replica's bulk load states and replica's bulk load context
• download_sst_files:
  • create local directory to store sst files temporarily
  • download metadata file synchronously (not implemented)
  • parse metadata structure from file (not implemented)
  • download sst files asynchronously (not implemented)
  • if download files succeed, verify sst files' metadata and update download progress (not implemented)
  • if download files failed or not pass verification, store error in _download_status, which will be reported to primary and meta server.

Function download_sst_files won't have unit test, but functions called by it such as do_download, parse_bulk_load_metadata have unit tests, will be added in further pull request.

New configurations

[replication]
+ max_concurrent_bulk_load_downloading_count = 5